Papers with pretrained models
“Is Whole Word Masking Always Better for Chinese BERT?”: Probing on Chinese Grammatical Error Correction (2022.findings-acl)
Copied to clipboard
| Challenge: | a Chinese model with whole word masking has no subword because each token is an atomic character. |
| Approach: | They propose to use whole word masking to mask all subwords corresponding to a word at once . they ask models to revise or insert tokens in a masked language modeling manner . |
| Outcome: | The proposed model performs better when one character is inserted or replaced . the model trained with standard character-level masking performs best when one token is masked . |
Attention is not not Explanation (D19-1)
Copied to clipboard
| Challenge: | Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. |
| Approach: | They propose to use a simple uniform-weights baseline, a variance calibration and a diagnostic framework to determine when/whether attention can be used as explanation in RNN models. |
| Outcome: | The proposed tests show that even reliable adversarial distributions don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability. |
A Self-Supervised Integration Method of Pretrained Language Models and Word Definitions (2023.findings-acl)
Copied to clipboard
| Challenge: | DefBERT integrates word definition modeling with word semantics in dictionaries. |
| Approach: | They propose a method that integrates pretrained models with word semantics in dictionaries. |
| Outcome: | The proposed method is effective on word similarity and short text classification tasks. |
More Discriminative Sentence Embeddings via Semantic Graph Smoothing (2024.eacl-short)
Copied to clipboard
| Challenge: | Text categorization is a natural language processing task that involves arranging texts into coherent groups based on their content. |
| Approach: | They propose to use semantic graph smoothing to enhance sentence embeddings from pretrained models to improve results for supervised and unsupervised document categorization tasks. |
| Outcome: | The proposed method improves sentences embeddings for supervised and unsupervised document categorization tasks. |
ViLMedic: a framework for research at the intersection of vision and language in medical AI (2022.acl-demo)
Copied to clipboard
Jean-benoit Delbrouck, Khaled Saab, Maya Varma, Sabri Eyuboglu, Pierre Chambon, Jared Dunnmon, Juan Zambrano, Akshay Chaudhari, Curtis Langlotz
| Challenge: | Multimodal medical AI is a growing field of interest, especially for tasks that involve multimodal data. |
| Approach: | They propose a vision-and-language medical library to improve multimodal medical predictions and enable new applications. |
| Outcome: | The vision-and-language medical library aims to improve reproducibility and speed up progress across medical AI . it contains a dozen implementations replicating state-of-the-art results on medical datasets . the library is extensible by researchers but also simple for practitioners . |
Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers (2026.acl-srw)
Copied to clipboard
| Challenge: | Large language models are difficult to reverse engineer because of their internal operations. |
| Approach: | They train small attention-only transformers on a symbolic version of the Indirect Object Identification task. |
| Outcome: | The proposed model with only two attention heads achieves perfect IOI accuracy despite lacking MLPs and normalization layers . |
Transformers: State-of-the-Art Natural Language Processing (2020.emnlp-demos)
Copied to clipboard
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, Alexander Rush
| Challenge: | Transformers is an open-source library that aims to open up advances in natural language processing to the wider machine learning community. |
| Approach: | they propose an open-source library that aims to open up advances in machine learning to the wider community. |
| Outcome: | Transformers is an open-source library with the goal of opening up these advances to the wider machine learning community. |
AdapterHub Playground: Simple and Flexible Few-Shot Learning with Adapters (2022.acl-demo)
Copied to clipboard
Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, Iryna Gurevych
| Challenge: | AdapterHub Playground is an open-access tool for researchers to use pretrained language models without writing a single line of code. |
| Approach: | They propose a tool which allows researchers to leverage pretrained models without writing a single line of code for a variety of NLP tasks. |
| Outcome: | The proposed model can be used for prediction, training and analysis of textual data without writing a single line of code. |
A Call for Clarity in Beam Search: How It Works and When It Stops (2024.lrec-main)
Copied to clipboard
| Challenge: | Empirical results show that a modified beam decoding implementation improves decoding performance of strong, neural language generation models. |
| Approach: | They propose a modification to a beam decoding implementation that generalizes the stopping criterion and provides flexibility to the depth of search. |
| Outcome: | The proposed method improves decoding performance of strong models on news text summarization and machine translation over diverse language pairs with negligible inference slowdown. |
Vision-Language Pretraining: Current Trends and the Future (2022.acl-tutorials)
Copied to clipboard
| Challenge: | Recent vision-language models are being used for downstream tasks that require large datasets and supervised datasets. |
| Approach: | They focus on recent vision-language pretraining paradigms and their strengths and shortcomings . they compare the different family of models used for vision- language pretraining . |
| Outcome: | This paper provides the background on image–language datasets, benchmarks, and modeling innovations before the multimodal pretraining area. |
MoExtend: Tuning New Experts for Modality and Task Extension (2024.acl-srw)
Copied to clipboard
| Challenge: | Existing instruction tuning methods for large language models (LLMs) are costly and difficult to implement. |
| Approach: | They propose a framework to streamline the modality adaptation and extension of Mixture-of-Experts (MoE) models. |
| Outcome: | The proposed framework enables rapid adaptation and extension to new modal data or tasks without tuning pretrained models. |
Trankit: A Light-Weight Transformer-based Toolkit for Multilingual Natural Language Processing (2021.eacl-demos)
Copied to clipboard
| Challenge: | Trankit is a lightweight, pre-trained toolkit for multilingual natural language processing. |
| Approach: | They propose a transformer-based toolkit for multilingual natural language processing that trains pipelines over 100 languages and 90 pretrained pipelines for 56 languages. |
| Outcome: | The proposed tool outperforms existing pipelines over sentence segmentation, part-of-speech tagging, morphological feature tabbing, and dependency parsing while maintaining competitive performance over tokenization, multi-word token expansion, and lemmatization over 90 Universal Dependencies treebanks. |
Better Quality Pre-training Data and T5 Models for African Languages (2023.emnlp-main)
Copied to clipboard
Akintunde Oladipo, Mofetoluwa Adeyemi, Orevaoghene Ahia, Abraham Owodunni, Odunayo Ogundepo, David Adelani, Jimmy Lin
| Challenge: | Existing web crawls have demonstrated quality issues for low-resource languages . Existing pretraining corpora have numerous quality issues . |
| Approach: | They propose to audit existing pretraining corpora to understand and rectify quality issues . they pretrain a new T5-based model and evaluate its performance on multiple tasks . |
| Outcome: | The proposed model outperforms existing pretrained models on four NLP tasks. |
PaddleSpeech: An Easy-to-Use All-in-One Speech Toolkit (2022.naacl-demo)
Copied to clipboard
Hui Zhang, Tian Yuan, Junkun Chen, Xintong Li, Renjie Zheng, Yuxin Huang, Xiaojie Chen, Enlei Gong, Zeyu Chen, Xiaoguang Hu, Dianhai Yu, Yanjun Ma, Liang Huang
| Challenge: | PaddleSpeech is an open-source speech toolkit that supports speech-to-text and text-to speech tasks. |
| Approach: | They describe the design philosophy and core architecture of PaddleSpeech to support several essential speech-to-text and text-to speech tasks. |
| Outcome: | The proposed framework achieves competitive or state-of-the-art performance on various speech datasets and implements the most popular methods. |
Investigating the Potential of Task Arithmetic for Cross-Lingual Transfer (2024.eacl-short)
Copied to clipboard
| Challenge: | Massively multilingual Transformer-based models (MMTs) can learn representations which have a degree of cross-lingual alignment despite being trained using purely unsupervised objectives. |
| Approach: | They propose a modular approach to cross-lingual transfer using task arithmetic . they show that modularity can be achieved even with full model fine-tuning . |
| Outcome: | The proposed approach shows strong performance on multilingual benchmarks encompassing both high-resource and low-resourced languages. |
Semantic Contrastive Adaptation for Multimodal Figurative Language Understanding (2026.acl-srw)
Copied to clipboard
| Challenge: | Existing models for understanding figurative language in images perform well on literal recognition but fail on multimodal figurativ benchmarks. |
| Approach: | They propose a model that adapts to idiomatic and figurative language using literal alignment bias rather than limited model capacity. |
| Outcome: | The proposed model generalizes across five idiom-rich languages despite being trained on English supervision. |
Stanza: A Python Natural Language Processing Toolkit for Many Human Languages (2020.acl-demos)
Copied to clipboard
| Challenge: | Existing tools that support only a few major languages are under-optimized for accuracy due to a focus on efficiency or use of less powerful models. |
| Approach: | They introduce a Python natural language processing toolkit that supports 66 languages . they train Stanza on 112 datasets and show it generalizes well on all languages compared to other tools . |
| Outcome: | The proposed toolkit performs well on 112 datasets and is compatible with the popular Java CoreNLP software. |
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually) (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining on self-supervised linguistic tasks is effective for learning features helpful for language understanding, but it requires more data to learn to prefer linguistic generalizations over surface ones. |
| Approach: | They propose a set of 20 ambiguous binary classification tasks to test whether a pretrained model prefers linguistic or surface generalizations. |
| Outcome: | The proposed model can learn to represent linguistic features with little pretraining data, but requires far more data to learn to prefer linguistic generalizations over surface ones. |
Pretrained Ensemble Learning for Fine-Grained Propaganda Detection (D19-50)
Copied to clipboard
| Challenge: | Propaganda detection is a reallife challenge that can affect how people understand news . |
| Approach: | They propose to use a manually annotated dataset to tackle the propaganda detection on sentence level classification task of NLP4IF 2019 workshop co-located with EMNLP-IJCNLP 2019 conference. |
| Outcome: | The proposed model is ranked in the first place with 68.8312 F1-score on the development dataset and in the sixth place with 61.3990 F1 score on the testing dataset. |
TeluguNER: Leveraging Multi-Domain Named Entity Recognition with Deep Transformers (2022.acl-srw)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is a successful and well-researched problem in English due to the availability of resources. |
| Approach: | They propose to use two annotated NER datasets for the Telugu language . they compare the finetuned Telugus model with the existing model in NER . |
| Outcome: | The proposed models outperform existing models on a large dataset of 38,363 sentences on telugu and other languages. |
Continual Pretraining on Encrypted Synthetic Data for Privacy-Preserving LLMs (2026.findings-eacl)
Copied to clipboard
| Challenge: | Existing methods to protect PII from training on small corpora are difficult to implement in real-world applications. |
| Approach: | They propose an entity-based framework that synthesizes encrypted training data to protect PII. |
| Outcome: | The proposed framework outperforms base models and ensures PII security on limited-scale datasets while exhibiting a modest performance gap compared to models trained on unencrypted synthetic data. |
Pretrain-KGE: Learning Knowledge Representation from Pretrained Language Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing knowledge graph embedding models suffer from limited knowledge representation due to sparse and noisy dataset annotations. |
| Approach: | They propose to use pretrained language models to enhance knowledge representation by leveraging world knowledge from pretrained models. |
| Outcome: | Extensive experiments show that the proposed framework can improve results over existing models. |
Universal Language Model Fine-tuning for Text Classification (P18-1)
Copied to clipboard
| Challenge: | Existing approaches to computer vision require task-specific modifications and training from scratch. |
| Approach: | They propose a method that can be applied to any task in NLP and propose to open-source it. |
| Outcome: | The proposed method outperforms the state-of-the-art on six text classification tasks, reducing error by 18-24% on majority of datasets. |
Zshot: An Open-source Framework for Zero-Shot Named Entity Recognition and Relation Extraction (2023.acl-demo)
Copied to clipboard
Gabriele Picco, Marcos Martinez Galindo, Alberto Purpura, Leopold Fuchs, Vanessa Lopez, Thanh Lam Hoang
| Challenge: | ZSL is a machine learning field that uses textual descriptions of entities or relations to perform tasks that are not seen during training. |
| Approach: | They propose a framework that allows researchers to compare state-of-the-art ZSL methods with standard benchmark datasets. |
| Outcome: | The proposed framework compares state-of-the-art methods with benchmark datasets and provides APIs for production under the standard SpaCy NLP pipeline. |
Farewell to Aimless Large-scale Pretraining: Influential Subset Selection for Language Model (2023.findings-acl)
Copied to clipboard
Xiao Wang, Weikang Zhou, Qi Zhang, Jie Zhou, SongYang Gao, Junzhe Wang, Menghan Zhang, Xiang Gao, Yun Wen Chen, Tao Gui
| Challenge: | Pretrained language models have achieved remarkable success in various natural language processing tasks. |
| Approach: | They propose to use end-task knowledge to select a tiny subset of pretraining corpus to influence performance. |
| Outcome: | The proposed model outperforms pretrained models on eight datasets covering four domains with 0.45% of the data and a three-orders-of-magnitude lower computational cost. |
Chemical Language Understanding Benchmark (2023.acl-industry)
Copied to clipboard
| Challenge: | CLUB datasets are used to facilitate NLP research in the chemical industry. |
| Approach: | They introduce a benchmark dataset called CLUB to facilitate NLP research in the chemical industry. |
| Outcome: | The CLUB datasets are a new benchmark dataset for NLP in the chemical industry. |
HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing (2021.eacl-demos)
Copied to clipboard
| Challenge: | Pretrained models have been taking the lead of many natural language processing benchmarks such as GLUE, but energy efficiency in the process of model training and inference becomes a critical bottleneck. |
| Approach: | They propose a multi-task energy efficiency benchmarking platform for responsible natural language processing that compares pretrained models’ energy efficiency from the perspectives of time and cost. |
| Outcome: | The proposed model improves on the fine-tuning efficiency of pretrained models from the perspectives of time and cost. |
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains (2021.findings-acl)
Copied to clipboard
| Challenge: | Large pre-trained models suffer from domain shift and are not optimal for specific domains. |
| Approach: | They propose a general approach to developing small, fast and effective pretrained models for specific domains by adapting off-the-shelf general pretrained model and performing task-agnostic knowledge distillation in target domains. |
| Outcome: | The proposed approach achieves better performance over the BERT BASE model in domain-specific tasks while 3.3 smaller and 5.1 faster than the BRT BASE. |
STAR: Spectral Truncation and Rescale for Model Merging (2025.naacl-short)
Copied to clipboard
| Challenge: | Model merging is an efficient way of obtaining a multi-task model from several pretrained models without further fine-tuning. |
| Approach: | They propose a model merging technique that aims at mitigating "merging conflicts" by truncating small components in the respective spectral spaces and then an automatic parameter rescaling scheme to retain the nuclear norm of the original matrix. |
| Outcome: | The proposed model outperforms baseline models on flan-T5 by 4.2% and is robust to hyperparamater choice. |
DOSE: Data Selection for Multi-Modal LLMs via Off-the-Shelf Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing data filtering methods are expensive because they are trained on the same data they are meant to screen. |
| Approach: | They propose to use off-the-shelf pretrained models that have never seen the target data to select training samples for larger and stronger multimodal models without task-specific training. |
| Outcome: | The proposed method can achieve comparable or even better results than those trained on the full dataset in standard VQA and math benchmarks. |
User-Centric Gender Rewriting (2022.naacl-main)
Copied to clipboard
| Challenge: | Existing systems that embed and amplify gender bias can still exhibit and exacerbate this problem. |
| Approach: | They propose a multi-step system that combines the positive aspects of rule-based and neural rewriting models to provide personalized outputs based on the users’ grammatical gender preferences. |
| Outcome: | The proposed system achieves 88.42 M2 F0.5 on a blind test set and improves over previous work on the first-person-only version of this task by 3.05 absolute increase in M2F0.5. |
Domain Generalisation of NMT: Fusing Adapters with Leave-One-Domain-Out Training (2022.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in neural machine translation (NMT) research have found NMT models sensitive to distribution shift and adversarial examples. |
| Approach: | They propose a leave-one-domain-out training strategy that learns to combine domain-specific parameters to avoid information leaking. |
| Outcome: | The proposed method outperforms baselines on three language pairs on average. |
Probing Critical Learning Dynamics of PLMs for Hate Speech Detection (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies on pretrained language models (PLMs) for hate speech detection have not investigated how their performance is affected by pretraining and finetuning. |
| Approach: | They propose to compare pretrained language models, evaluate their seed robustness, finetuning settings, and the impact of pretraining data collection time. |
| Outcome: | The proposed models show that they are more robust than other models and that they have a better chance of performing better than domain-specific models. |
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code (2025.coling-industry)
Copied to clipboard
Taishi Nakamura, Mayank Mishra, Simone Tedeschi, Yekun Chai, Jason T. Stillerman, Felix Friedrich, Prateek Yadav, Tanmay Laud, Vu Minh Chien, Terry Yue Zhuo, Diganta Misra, Ben Bogin, Xuan-Son Vu, Marzena Karpinska, Arnav Varma Dantuluri, Wojciech Kusa, Tommaso Furlanello, Rio Yokota, Niklas Muennighoff, Suhas Pai, Tosin Adewumi, Veronika Laippala, Xiaozhe Yao, Adalberto Barbosa Junior, Aleksandr Drozd, Jordan Clive, Kshitij Gupta, Liangyu Chen, Qi Sun, Ken Tsui, Nour Moustafa-Fahmy, Nicolo Monti, Tai Dang, Ziyang Luo, Tien-Tung Bui, Roberto Navigli, Virendra Mehta, Matthew Blumberg, Victor May, Hiep Nguyen, Sampo Pyysalo
| Challenge: | Pretrained language models are integral part of AI applications, but their high computational cost limits accessibility. |
| Approach: | They evaluate Aurora-M, a 15B parameter multilingual open-source model trained on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
| Outcome: | The proposed model outperforms existing models on English, Finnish, Hindi, Japanese, Vietnamese, and code. |
Disentangling Representations of Text by Masking Transformers (2021.emnlp-main)
Copied to clipboard
| Challenge: | Large pretrained models such as BERT encode a range of features into monolithic vectors, providing strong predictive accuracy across downstream tasks. |
| Approach: | They explore whether it is possible to learn disentangled representations by identifying existing subnetworks within pretrained models that encode distinct, complementary aspects. |
| Outcome: | The proposed method disentangles sentiment from genre in movie reviews, toxicity from dialect in Tweets, and syntax from semantics. |
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches require substantial adaptation of pretrained language models for vision-language reasoning tasks. |
| Approach: | They propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. |
| Outcome: | The proposed framework outperforms the Flamingo model on VQAv2 and GQA by 8.5%. |
Auto-Debias: Debiasing Masked Language Models with Automated Biased Prompts (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods to mitigate human-like biases in pretrained language models are based on external corpora and require a distribution alignment loss to mitigate them. |
| Approach: | They propose an automatic method to mitigate biases in pretrained language models by searching for biased prompts such that cloze-style completions are the most different with respect to different demographic groups. |
| Outcome: | The proposed method reduces biases in pretrained language models, including gender and racial bias, and improves fairness of the models. |
DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Sentence embeddings are an important component of many natural language processing systems. |
| Approach: | They propose a self-supervised objective for learning universal sentence embeddings that does not require labelled training data. |
| Outcome: | The proposed approach closes the performance gap between unsupervised and supervised pretraining for universal sentence encoders. |
Does BERT Pretrained on Clinical Notes Reveal Sensitive Data? (2021.naacl-main)
Copied to clipboard
| Challenge: | Pretraining large (masked) language models over EHR data has yielded consistent performance gains across tasks. |
| Approach: | They propose to use large Transformers to release pretraining models over EHRs . they propose to recover patient names and conditions associated with them . |
| Outcome: | The proposed models recover patient names and conditions associated with patients . the proposed models share the model parameters for use by other researchers . |
Improving Pretrained Models for Zero-shot Multi-label Text Classification through Reinforced Label Hierarchy Reasoning (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing zero-shot learning methods for multi-label text classification mostly learn a matching model between the feature space of text and the label space. |
| Approach: | They propose to use a graph encoder to incorporate label hierarchies to learn effective label representations on the zero-shot multi-label text classification problem. |
| Outcome: | The proposed approach outperforms previous non-pretrained methods on the zero-shot multi-label text classification task. |
Measuring Progress in Fine-grained Vision-and-Language Understanding (2023.acl-long)
Copied to clipboard
| Challenge: | X-VLM models lack "fine-grained" understanding of relationships, verbs and numbers in images . pretraining on large-scale image–text data from the Web has facilitated rapid progress on many vision-and-language tasks . |
| Approach: | They investigate models that outperform other baselines on fine-grained data . they highlight importance of novel losses and rich data sources for learning fine-grain skills . |
| Outcome: | The proposed model outperforms baseline models on four fine-grained benchmarks . the model outpersforms other baseline models and even degrades performance . |
Logic Against Bias: Textual Entailment Mitigates Stereotypical Sentence Reasoning (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent studies show that textual entailment learning reduces social biases in pretrained sentence encoders. |
| Approach: | They compare pretrained sentence encoders with textual entailment models that learn language logic for downstream language understanding tasks. |
| Outcome: | The proposed models outperform models with lower bias without debiasing processes on stereotype, profession, and emotion bias tests. |
Language Model Adaption for Reinforcement Learning with Natural Language Action Space (2024.acl-long)
Copied to clipboard
| Challenge: | Previous research has focused on reducing the size of the natural language action space due to the combinatorial nature of the language. |
| Approach: | They propose mutual-information regularized policy optimization to reduce the action space by dynamically adjusting the prior provided by the pretrained model. |
| Outcome: | The proposed method improves monotonically on the mutual-information regularized RL objective. |
SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in Fine-tuned Source Code Models (2024.findings-naacl)
Copied to clipboard
| Challenge: | Large datasets are increasingly available for pre-training source code models, but obtaining representative training data that fully covers the code distribution for specific downstream tasks remains challenging due to the task-specific nature and limited labeling resources. |
| Approach: | They propose a systematic approach that simulates various OOD scenarios along different dimensions of source code data properties and investigates model behavior under different fine-tuning methodologies. |
| Outcome: | The proposed approach simulates various OOD scenarios along different dimensions of source code data properties and exposes multiple failure modes attributed to OOD generalization issues. |
RewardBench: Evaluating Reward Models for Language Modeling (2025.findings-naacl)
Copied to clipboard
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, Hannaneh Hajishirzi
| Challenge: | Evaluating reward models presents an opportunity to understand the opaque technologies used for alignment of language models. |
| Approach: | They present a benchmark dataset and code-base for evaluation of reward models . they use prompt-chosen-rejected trios to benchmark how they perform on queries . |
| Outcome: | The proposed dataset compares RMs with other models on a set of questions. |
AfroMT: Pretraining Strategies and Reproducible Benchmarks for Translation of 8 African Languages (2021.emnlp-main)
Copied to clipboard
| Challenge: | Existing reproducible benchmarks for machine translation are limited to high-resource or well-represented languages. |
| Approach: | They propose to use AfroMT to develop a reproducible machine translation benchmark for eight widely spoken African languages and a suite of analysis tools to take into account their unique properties. |
| Outcome: | The proposed benchmarks show significant improvements when pretraining on 11 languages, with gains of up to 2 BLEU points over strong baselines. |
Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models (2026.acl-srw)
Copied to clipboard
| Challenge: | Model calibration is commonly used as a proxy for reliability, but low calibration error does not necessarily imply reliable decision rules. |
| Approach: | They investigate the calibration error gap in generative language models using the TOFU benchmark and attribution-based shortcut detection. |
| Outcome: | The proposed model calibrations achieve low calibration error compared to pretrained models and retain low calibration despite reduced accuracy on the forget split. |
Pretrained Models for Multilingual Federated Learning (2022.naacl-main)
Copied to clipboard
| Challenge: | Federated Learning (FL) is a machine learning technique that trains a model across multiple distributed clients holding local data samples, without ever storing client data in a central location. |
| Approach: | They propose to use pretrained models to study three multilingual language tasks . they also examine impact of non-IID text on FL in naturally occurring data . |
| Outcome: | The proposed methods perform better than centralized learning even when using non-IID partitioning. |
Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods (2024.findings-eacl)
Copied to clipboard
| Challenge: | Transfer learning methods have shown that reusing non-task-specific knowledge can speed up task-specific learning in the latter. |
| Approach: | They propose to port whole functional modules that encode task-specific knowledge from one model to another using parameter-efficient finetuning techniques. |
| Outcome: | The proposed methods outperform the two alternatives, but there are differences between the two. |
Representation Learning for Conversational Data using Discourse Mutual Information Maximization (2022.naacl-main)
Copied to clipboard
Bishal Santra, Sumegh Roychowdhury, Aishik Mandal, Vasu Gurram, Atharva Naik, Manish Gupta, Pawan Goyal
| Challenge: | Existing language modeling pretraining objectives do not take structural information of conversational text into account. |
| Approach: | They propose a structure-aware Mutual Information based loss-function DMI for training dialog-representation models that captures the inherent uncertainty in response prediction. |
| Outcome: | The proposed model outperforms strong baseline models on nine diverse tasks. |
Multi-Modal Bias: Introducing a Framework for Stereotypical Bias Assessment beyond Gender and Race in Vision–Language Models (2023.eacl-main)
Copied to clipboard
| Challenge: | Recent advances in self-supervised training have led to a new class of pretrained vision–language models. |
| Approach: | They propose a visual and textual bias benchmark to assess bias in self-supervised multimodal models using 3,800 images and phrases from 14 population subgroups. |
| Outcome: | The proposed model shows that it favors certain groups while maintaining the accuracy of the model. |
Describing Images Fast and Slow: Quantifying and Predicting the Variation in Human Signals during Visuo-Linguistic Processes (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing models of visuo-linguistic variation are weak to moderately trained to capture such a variation in visual outputs. |
| Approach: | They use a corpus of Dutch image descriptions with eye-tracking data to investigate the nature of the variation in visuo-linguistic signals. |
| Outcome: | The proposed model lacks biases about what makes a stimulus complex for humans and what leads to variations in human outputs. |
Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to cross-lingual text classification require task-specific training data in high-resource sources . labeling cost, task characteristics, and privacy concerns can hinder the use of cross-linguistic training . |
| Approach: | They propose a dictionary-based heterogeneous graph (DHGNet) that uses bilingual dictionaries for task-independent word embeddings. |
| Outcome: | The proposed method outperforms pretrained models even though it does not access to large corpora. |
Simple Local Attentions Remain Competitive for Long-Context Tasks (2022.naacl-main)
Copied to clipboard
Wenhan Xiong, Barlas Oguz, Anchit Gupta, Xilun Chen, Diana Liskovich, Omer Levy, Scott Yih, Yashar Mehdad
| Challenge: | Existing models for NLP tasks require long text sequences beyond the length limit of pretrained models. |
| Approach: | They propose to pretrain large-size NLP models using the same long-doc corpus and fine tune them for real-world long-context tasks. |
| Outcome: | The proposed models can perform better under standard pretraining paradigms than longformer and Longformer. |
Localization vs. Semantics: Visual Representations in Unimodal and Multimodal Models (2024.eacl-long)
Copied to clipboard
| Challenge: | Existing vision-and-language models perform better on multimodal tasks, but there is little understanding of how multimodal learning can help visual representations. |
| Approach: | They conduct a probing analysis of visual representations in existing vision-and-language models and vision-only models by probing on a broad range of tasks. |
| Outcome: | The proposed model improves vision-and-language models on label and attribute prediction tasks while vision-only models are stronger on dense prediction tasks. |
Using Similarity Measures to Select Pretraining Data for NER (N19-1)
Copied to clipboard
| Challenge: | Existing studies on how to select appropriate data to pretrain word vectors or LMs are lacking. |
| Approach: | They propose to quantify aspects of similarity between pretraining and target data. |
| Outcome: | The proposed measures are good predictors of the usefulness of pretrained models for Named Entity Recognition over 30 data pairs. |
OpenHands: Making Sign Language Recognition Accessible with Pose-based Pretrained Models across Languages (2022.acl-long)
Copied to clipboard
| Challenge: | a new study examines the performance of pretraining for sign language recognition in low-resource settings. |
| Approach: | They propose using pose extracted through pretrained models as the standard modality of data to reduce training time and enable efficient inference. |
| Outcome: | The proposed model reduces training time and allows efficient inference in sign languages. |
MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMs (2024.acl-long)
Copied to clipboard
| Challenge: | Existing models have demonstrated outstanding capabilities in mathematical reasoning, but there is a performance gap between open-source models and closed-source ones. |
| Approach: | They propose a method for generating diverse and reliable math problems by leveraging the ground-truth solutions of the seed data. |
| Outcome: | The proposed model outperforms open-source models across five representative mathematical reasoning datasets. |
Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that domain-specific BERT models can be improved when in-domain data is used for pretraining. |
| Approach: | They propose to use Twitter and forum text as pretraining sources for two BERT models and use similarity measures to nominate in-domain data for pretraining. |
| Outcome: | The proposed method can be used to improve performance on downstream tasks by using in-domain data. |
Influence Scores at Scale for Efficient Language Data Sampling (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent studies have shown that ML models can be fine-tuned on as much data as possible without degradation in performance metrics. |
| Approach: | They evaluate the applicability of influence scores in language classification tasks by random sampling and stress-testing one of the scores. |
| Outcome: | The proposed model can be fine-tuned on 50% of the original data without degradation in performance metrics. |
Systematic Investigation of Strategies Tailored for Low-Resource Settings for Low-Resource Dependency Parsing (2023.eacl-main)
Copied to clipboard
| Challenge: | Several strategies have been proposed to enhance performance in low-resource scenarios. |
| Approach: | They propose to use 5 low-resource strategies for dependency parsing for multiple languages . they use ensembled approach on 7 UD low-rsource languages based on their results . |
| Outcome: | The proposed approach improves on a low-resource language Sanskrit. |
Dim Wihl Gat Tun: The Case for Linguistic Expertise in NLP for Under-Documented Languages (2022.findings-acl)
Copied to clipboard
Clarissa Forbes, Farhan Samir, Bruce Oliver, Changbing Yang, Edith Coates, Garrett Nicolai, Miikka Silfverberg
| Challenge: | Recent progress in NLP is driven by pretrained models leveraging massive datasets. |
| Approach: | They argue that IGT data can be leveraged provided target language expertise is available and that it can be used to create effective models. |
| Outcome: | The proposed model can be leveraged provided that target language expertise is available. |
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks (N18-1)
Copied to clipboard
| Challenge: | Existing approaches to learn to do syntactically controlled paraphrase generation are limited . lexical, pragmatic, and syntaktic variation can hurt generalization of models trained on them . |
| Approach: | They propose a new approach for learning to do syntactically controlled paraphrase generation using a parser. |
| Outcome: | The proposed model generates paraphrases that follow their target specifications without decreasing paraphrase quality compared to baseline models . it improves the robustness of the models to syntactic variation when used to augment training data. |
Hierarchical Pretraining on Multimodal Electronic Health Records (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing pretraining models on EHR data are too specific, limiting their transferability. |
| Approach: | They propose a general, unified pretraining framework for hierarchically multimodal EHR data that can be used to train models on a large dataset before fine-tuning it on 'upstream' tasks. |
| Outcome: | The proposed model performs on eight downstream tasks spanning three levels and compares with baselines on 18 different tasks. |
On Faithfulness and Factuality in Abstractive Summarization (2020.acl-main)
Copied to clipboard
| Challenge: | Existing conditional text generation models produce unfaithful and unfaithed summaries . current models accomplish a high level of fluency and coherence . |
| Approach: | They propose to use pretrained models for document summarization to better understand hallucinations . they find that textual entailment measures better correlate with faithfulness . |
| Outcome: | The proposed models generate faithful and factual summaries as evaluated by humans. |
BiKT: Enabling Bidirectional Knowledge Transfer Between Pretrained Models and Sequential Downstream Tasks (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing frameworks adapt from initial pretrained model to each downstream task directly, but ignore sequential nature of downstream tasks and feedback effect on pretrained models. |
| Approach: | They propose a framework to enable bidirectional knowledge transfer between pretrained models and downstream tasks in rounds. |
| Outcome: | The proposed framework improves on 9 GLUE datasets and 6 SuperGLUEs. |
Extending Input Contexts of Language Models through Training on Segmented Sequences (2024.findings-naacl)
Copied to clipboard
| Challenge: | Effectively training languages models on long sequences poses many technical challenges. |
| Approach: | They propose a method for extending positional embeddings by sub-sampling segments from long inputs while maintaining their original position. |
| Outcome: | The proposed method extends the input con-text size of pretrained models without any changes in the model's memory and memory costs. |
To Pretrain or Not to Pretrain: Examining the Benefits of Pretrainng on Resource Rich Tasks (2020.acl-main)
Copied to clipboard
| Challenge: | Existing studies on pretraining NLP models with variants of Masked Language Model (MLM) objectives have shown that the number of training samples used in the downstream task is limited. |
| Approach: | They propose to use MLM objectives to pretrain NLP models with variants of Masked Language Model (MLM) objectives to improve accuracy on downstream tasks. |
| Outcome: | The proposed model can reach a diminishing return point as the supervised data size increases significantly. |
RASAT: Integrating Relational Structures into Pretrained Seq2Seq Model for Text-to-SQL (2022.emnlp-main)
Copied to clipboard
Jiexing Qi, Jingyao Tang, Ziwei He, Xiangpeng Wan, Yu Cheng, Chenghu Zhou, Xinbing Wang, Quanshi Zhang, Zhouhan Lin
| Challenge: | Experimental results show RASAT can leverage a variety of relational structures while inheriting the pretrained parameters from the T5 model. |
| Approach: | They propose a Transformer seq2seq architecture augmented with relation-aware self-attention that leverages relational structures while inheriting pretrained parameters from the T5 model. |
| Outcome: | The proposed model can leverage relational structures while inheriting pretrained parameters from the T5 model effectively. |
Few-Shot Question Answering by Pretraining Span Selection (2021.acl-long)
Copied to clipboard
| Challenge: | Pretraining models with recurring span selection are effective, but perform poorly in a few-shot setting. |
| Approach: | They propose recurring span selection scheme that asks model to select correct span in passage with multiple sets of recurring recurrings. |
| Outcome: | The proposed model achieves 72.7 F1 on multiple benchmarks while maintaining competitive performance in the high-resource setting. |
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2023.eacl-main)
Copied to clipboard
| Challenge: | Pre-training/fine-tuning of pre-training models has become more expensive and resource-hungry. |
| Approach: | They propose a low-rank adaptation technique that trains LoRA blocks for a range of ranks instead of a single rank. |
| Outcome: | The proposed method trains LoRA blocks for a range of ranks instead of a single rank . it can train dynamic search-free models with DyLoRA at least 4 to 7 times faster than LoRA . |
Improving Reward Models with Synthetic Critiques (2025.findings-naacl)
Copied to clipboard
| Challenge: | a recent study shows that reward models overfit on superficial features, hindering generalization performance . prevailing approach to training preference-based reward models presents several challenges . |
| Approach: | They propose a method that uses synthetic natural language critiques to provide additional feedback to large language models. |
| Outcome: | The proposed approach improves performance and data efficiency of RMs initialized from different pretrained models, reducing the reliance on costly human annotations. |
Exploring Unsupervised Pretraining Objectives for Machine Translation (2021.findings-acl)
Copied to clipboard
| Challenge: | Unsupervised cross-lingual pretraining has significantly reduced the need for large parallel data. |
| Approach: | They compare unsupervised cross-lingual pretraining with masking and reconstructing inputs in the decoder to produce real sentences. |
| Outcome: | The proposed methods produce inputs resembling real (full) sentences, by reordering and replacing words based on their context. |
Spectral Filters, Dark Signals, and Attention Sinks (2024.acl-long)
Copied to clipboard
| Challenge: | Recent work assigns a central role to the model's residual stream as the shared communication channel between model components. |
| Approach: | They propose a quantitative extension of the logit lens approach by partitioning the embedding and unembedding matrices into bands and spectral filters on intermediate representations. |
| Outcome: | The proposed model can suppress the tail end of the embedding spectrum, but it is not able to suppress large parts of the spectrum. |
Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent? (2025.findings-emnlp)
Copied to clipboard
| Challenge: | Cross-lingual consistency should be considered to assess cross-lingual transferability, maintain factuality of model knowledge across languages, and preserve parity of language model performance. |
| Approach: | They examine pretrained and tuned models with code-mixed coreferential statements that convey identical knowledge across languages. |
| Outcome: | The proposed model shows different levels of consistency in multilingual models, subject to language families, linguistic factors, scripts, and bottlenecks on a particular layer. |
VD-BERT: A Unified Vision and Dialog Transformer with BERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Prior work focused on attention mechanisms to model complex interactions in visual dialog . a new framework for visual dialog is based on pretrained BERT language models . |
| Approach: | They propose a framework for a vision-dialog Transformer that leverages pretrained BERT language models for Visual Dialog tasks. |
| Outcome: | The proposed framework achieves the top position on the visual dialog leaderboard without pretraining on external vision-language data. |
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (P19-1)
Copied to clipboard
| Challenge: | Term memory networks (RNNs) are difficult to optimize due to gradient vanishing and explosion. |
| Approach: | They propose a neural architecture Transformer-XL that enables learning dependency beyond a fixed length without disrupting temporal coherence. |
| Outcome: | The proposed method improves state-of-the-art performance on short and long sequences and generates coherent, novel text articles with thousands of tokens. |
Integrating Task Specific Information into Pretrained Language Models for Low Resource Fine Tuning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing pretrained language models are agnostic to downstream information and can overfit when fine-tuned with low resource datasets. |
| Approach: | They integrate label information as a task-specific prior into the self-attention component of pretrained BERT models. |
| Outcome: | Experiments on benchmarks and real-word datasets show that the proposed approach can improve the performance of pretrained models when fine-tuned with small datasets. |
Efficient Transformer-based Large Scale Language Representations using Hardware-friendly Block Structured Pruning (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained large-scale language models have been criticized for their limited weight storage and computational speed on hardware platforms. |
| Approach: | They propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. |
| Outcome: | The proposed model achieves 5.0x accuracy on GLUE benchmarks and 1.79x compression rate on DistilBERT. |
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pretrained language models suffer from poor generalization . however, they add a perturbation to each model parameter equally, which is sub-optimal . |
| Approach: | They propose a sharpness-aware minimization optimization procedure that introduces a Fisher mask to improve the efficiency of SAM. |
| Outcome: | The proposed method outperforms the vanilla sharpness-aware minimization method on GLUE and SuperGLUE benchmarks. |
Layer or Representation Space: What Makes BERT-based Evaluation Metrics Robust? (2022.coling-1)
Copied to clipboard
| Challenge: | Recent embedding-based evaluation metrics for text generation are based on measuring correlation with human evaluations on standard benchmarks. |
| Approach: | They examine the robustness of BERTScore, one of the most popular embedding-based metrics for text generation. |
| Outcome: | The embedding-based metrics that have the highest correlation with human evaluations on a standard benchmark can have the lowest correlation if the amount of input noise or unknown tokens increases. |
Factual Dialogue Summarization via Learning from Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Existing models generate fluent and coherent summaries, but inconsistencies can be found in generated summary. |
| Approach: | They propose to use symbolic knowledge distillation to improve the factual consistency of smaller pretrained models for dialogue summarization. |
| Outcome: | The proposed model outperforms baseline models in BART, PEGASUS, and Flan-T5 in factual consistency and accuracy. |
Understanding Attention for Vision-and-Language Tasks (2022.coling-1)
Copied to clipboard
| Challenge: | Attention mechanism has been used in Vision-and-Language (VL) tasks to bridge the semantic gap between visual and textual clues. |
| Approach: | They conduct a comprehensive analysis on understanding the role of attention alignment by looking into attention score calculation methods and checking how it represents the visual region’s and textual token’s significance for the global assessment. |
| Outcome: | The attention score calculation methods represent visual region’s and textual token’s significance for the global assessment. |
How Good are LLM-based Rerankers? An Empirical Analysis of State-of-the-Art Reranking Models (2025.findings-emnlp)
Copied to clipboard
| Challenge: | a systematic and comprehensive empirical evaluation of state-of-the-art reranking methods is presented. |
| Approach: | They evaluate 22 reranking methods including 40 variants across established benchmarks . primary goal is to determine whether performance disparity exists between LLM-based reranters and lightweight counterparts based on novel queries . |
| Outcome: | The proposed methods perform better on familiar queries than lightweight models, the authors show . |
Model Surgery: Modulating LLM’s Behavior Via Simple Parameter Editing (2025.naacl-long)
Copied to clipboard
| Challenge: | Current approaches for detoxification or preventing jailbreaking involve fine-tuning billions of parameters through gradient descent with substantial computational cost. |
| Approach: | They propose to use supervised fine-tuning and Reinforcement Learning from human feedback to modify LLMs' behavior by directly editing a small subset of parameters. |
| Outcome: | Experiments show that editing a small subset of parameters can modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreak, with only inference-level computational resources. |
Are “Undocumented Workers” the Same as “Illegal Aliens”? Disentangling Denotation and Connotation in Vector Spaces (2020.emnlp-main)
Copied to clipboard
| Challenge: | popular pretrained models encode both denotation and connotation as one entangled representation . a researcher using a pretrained representation can confuse words with connotations . |
| Approach: | They propose a nerual netowrk that decomposes a pretrained representation as independent denotation and connotation representations. |
| Outcome: | The proposed model improves document rankings by comparing denotation and connotation representations with extrinsic representations. |
Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent research has focused on using synthetic data for grammatical error correction . lack of annotated training data hinders progress in the field . |
| Approach: | They propose auxiliary tasks that exploit alignment between original and corrected sentences . they propose a sequence-to-sequence problem and perform multi-task training . |
| Outcome: | The proposed auxiliary tasks outperform the best models with a BART-based model on 11B parameters. |
SciBERT: A Pretrained Language Model for Scientific Text (D19-1)
Copied to clipboard
| Challenge: | SciBERT is a pretrained language model based on BERT to improve performance on scientific NLP tasks. |
| Approach: | They propose a pretrained language model based on BERT to improve NLP performance . they evaluate on sequence tagging, sentence classification and dependency parsing . |
| Outcome: | The proposed model improves on sequence tagging, sentence classification and dependency parsing tasks with datasets from a variety of scientific domains. |
Switch Point biased Self-Training: Re-purposing Pretrained Models for Code-Switching (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Code-switching (CS) is a phenomenon of switching between multiple languages . current models cannot handle CS due to lack of annotated data and limited resources. |
| Approach: | They propose a self-training method to repurpose existing models using a switch-point bias by leveraging unannotated data to reduce the gap between the switch point performance and retain overall performance on two distinct language pairs. |
| Outcome: | The proposed model reduces the gap between the switch point performance while retaining the overall performance on two distinct language pairs. |
How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers? (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Task-agnostic data augmentations have proven widely effective in computer vision, even on pretrained models. |
| Approach: | They examine the effects of two types of task-agnostic data augmentation on pretrained transformers using 5 classification tasks and 6 datasets. |
| Outcome: | The proposed techniques improve performance on 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers. |
CTRLsum: Towards Generic Controllable Text Summarization (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing summarization systems produce generic summaries that are disconnected from users’ preferences and expectations. |
| Approach: | They propose a generic framework to control generated summaries through a set of keywords. |
| Outcome: | The proposed framework is comparable or better than strong pretrained systems on three domains of summarization datasets and five control tasks. |
Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing studies on grammatical error correction (GEC) in morphologically rich languages have been limited due to data scarcity and language complexity. |
| Approach: | They propose to use Arabic GEC to improve performance across three datasets . they define Arabic grammatical error detection task as auxiliary input . |
| Outcome: | The proposed models achieve SOTA results on two Arabic GEC shared task datasets and establish a strong benchmark on a recently created dataset. |
Analyzing Redundancy in Pretrained Transformer Models (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that transformer-based deep NLP models are over-parameterized and do not require all the representational power lent by the rich architectural choices during inference. |
| Approach: | They define a notion of Redundancy and propose a feature-based transfer learning procedure which maintains 97% performance while using at-most 10% of the original neurons. |
| Outcome: | The proposed model maintains 97% performance while using 10% of the original neurons. |
PAKTON: A Multi-Agent Framework for Question Answering in Long Legal Agreements (2025.emnlp-main)
Copied to clipboard
| Challenge: | Contract review is a complex and time-intensive task that typically requires legal expertise. |
| Approach: | a new open-source contract review framework is designed to handle complexities of contract analysis . PAKTON is a retrieval-augmented generation framework with plug-and-play capabilities . |
| Outcome: | The open-source framework outperforms models in predictive accuracy, retrieval performance, explainability, completeness, and grounded justifications. |
PROST: Physical Reasoning about Objects through Space and Time (2021.findings-acl)
Copied to clipboard
| Challenge: | a new dataset is available to test pretraining of physical reasoning models . state-of-the-art models are inadequate at reasoning about physical interactions, authors say . |
| Approach: | They present a dataset that contains 18,736 multiple-choice questions from 14 templates . they propose to use the dataset to probe both causal and masked language models . |
| Outcome: | The proposed dataset contains 18,736 multiple-choice questions covering 10 physical reasoning concepts. |
Long Document Ranking with Query-Directed Sparse Transformer (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to document ranking require long documents to be broken to fit in pretrained models. |
| Approach: | They propose a Query-Directed Sparse attention model that induces IR-axiomatic structures in transformer self-attention. |
| Outcome: | The proposed model enforces the principle properties desired in ranking while also enjoying efficiency from sparsity. |
Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings (2020.acl-main)
Copied to clipboard
| Challenge: | Contextualized representations have become the default for downstream NLP applications. |
| Approach: | They propose a method for converting from contextualized representations to static lookup-table embeddings and apply it to 5 popular pretrained models and 9 sets of pretrained weights. |
| Outcome: | The proposed methods show that pooling over many contexts significantly improves representational quality under intrinsic evaluation. |
DocNLI: A Large-scale Dataset for Document-level Natural Language Inference (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing studies focus on sentence-level inference, which limits its application in downstream NLP problems. |
| Approach: | They propose to construct a large-scale dataset for document-level NLI that can be used to study NLP problems. |
| Outcome: | The proposed model performs well on popular sentence-level benchmarks and generalizes well to out-of-domain NLP tasks that rely on inference at document granularity. |
Transfer Learning for Sequence Generation: from Single-source to Multi-source (2021.acl-long)
Copied to clipboard
| Challenge: | Recent studies have shown that pretrained models are effective for low-resource downstream tasks. |
| Approach: | They propose a two-stage finetuning method to transfer pretrained models to MSG tasks by concatenating multiple sources into a single long sequence. |
| Outcome: | The proposed model outperforms baselines on the WMT17 APE task and multi-source translation task using the WTM14 test set. |
Intermediate-Task Transfer Learning with Pretrained Language Models: When and Why Does It Work? (2020.acl-main)
Copied to clipboard
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, Samuel R. Bowman
| Challenge: | Unsupervised pretraining has recently pushed the state of the art on many natural language understanding tasks. |
| Approach: | They perform a large-scale survey on a pretrained RoBERTa model with 110 intermediate-target task combinations and 25 probing tasks to reveal the specific skills that drive transfer. |
| Outcome: | The proposed model is trained on 110 intermediate-target task combinations and compared with 25 probing tasks to reveal the specific skills that drive transfer. |
Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generation (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to pretrain language models are limited by one-size-fits-all vocabulary . embeddings of mismatch tokens can be efficiently initialized in downstream tasks . |
| Approach: | They propose to extend pretrain-finetune pipeline with an embedding transfer step . plug-and-play embeddable generator is introduced to generate any input token . |
| Outcome: | The proposed approach allows for more efficient and better performed NLG models. |
VLASCD: A Visual Language Action Model for Simultaneous Chatting and Decision Making (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent large-scale pretrained models are built upon a multi-input single-output paradigm . tasks compete for a shared output channel, creating mutual exclusion effects . |
| Approach: | They propose a multi-input single-output (MISO) paradigm for large pretrained models . they propose unified training framework that enables concurrent multi-task outputs . |
| Outcome: | Experiments on autonomous driving platform show that MIMO-VLA outperforms state-of-the-art models in MIMO settings. |
Training Question Answering Models From Synthetic Data (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing work on question and answer generation aims to improve question answering models given limited amount of labeled data. |
| Approach: | They synthesize questions and answers from a synthetic text corpus generated by an 8.3 billion parameter GPT-2 model and achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. |
| Outcome: | The proposed model achieves higher accuracy than the SQUAD1.1 training set questions using synthetic questions and answers than the training set question. |
Supervised and Unsupervised Probing of Shortcut Learning: Case Study on the Emergence and Evolution of Syntactic Heuristics in BERT (2025.findings-acl)
Copied to clipboard
| Challenge: | Contemporary language models (LMs) rely on shortcut learning, using superficial cues that are spuriously correlated with labels. |
| Approach: | They propose to use syntactic heuristics to learn shortcuts in BERT when performing a task in Natural Language Understanding to investigate where these shortcuts emerge, how they evolve and how they impact the latent knowledge of the LM. |
| Outcome: | The proposed model rely on syntactic heuristics when performing a task in Natural Language Understanding. |
Improving Zero-Shot Multilingual Text Generation via Iterative Distillation (2022.coling-1)
Copied to clipboard
| Challenge: | Existing approaches to generalize multilingual dialogue systems to multilingual settings often make assumptions about data availability. |
| Approach: | They propose to transfer inductive biases for target languages learned by pretrained teacher models to student models via sequence-level knowledge distillation. |
| Outcome: | The proposed method performs well on the multiATIS++ benchmark, and is comparable to human annotations in both slot F1 and intent accuracy. |
Parameter-Efficient Tuning Makes a Good Classification Head (2022.emnlp-main)
Copied to clipboard
| Challenge: | In recent years, pretrained models revolutionized the paradigm of natural language understanding . but the final-layer output of the backbone, i.e. the input of the classification head, will change greatly during finetuning . |
| Approach: | They propose to append a randomly initialized classification head after the pretrained backbone and finetune the whole model. |
| Outcome: | The proposed classification head can be replaced with the randomly initialized heads for a stable performance gain. |
A Sequence-to-Structure Approach to Document-level Targeted Sentiment Analysis (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Aspect-based sentiment analysis (ABSA) has received wide attention in NLP for nearly two decades . previous studies focused on sentence-level ABSA, but document-level research has not received enough attention. |
| Approach: | They propose a Sequence-to-Structure approach to address the document-level targeted sentiment analysis task, which aims to extract the opinion targets consisting of multi-level entities from a review document and predict their sentiments. |
| Outcome: | The proposed approach outperforms baselines on six domains on the document-level targeted sentiment analysis task. |
What do Large Language Models Learn beyond Language? (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Pretraining on text confers models with useful ‘inductive biases’ for non-linguistic reasoning. |
| Approach: | They investigate whether pre-training on text confers these models with helpful ‘inductive biases’ for non-linguistic reasoning. |
| Outcome: | The proposed models outperform non-pretrained models on 19 non-linguistic tasks and show that they retain inductive biases even when training on multi-lingual text and computer code. |
Gender Biases in Automatic Evaluation Metrics for Image Captioning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Pretrained evaluation metrics can perpetuate and amplify biases, causing inability to differentiate between biased and unbiased generations. |
| Approach: | They conduct a systematic study of gender biases in image captioning tasks . they show that pretrained models perpetuate and amplify biase . |
| Outcome: | The proposed model-based evaluation metrics have shown good correlations with human judgments in language generation tasks. |
Assessing News Thumbnail Representativeness: Counterfactual text can enhance the cross-modal matching ability (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing models for visual and language processing struggle to match news actors’ visual and textual appearances. |
| Approach: | They propose a method that generates counterfactual news thumbnail images and text pairs to assess whether a news thumbnail image represents the actors discussed in the news text. |
| Outcome: | The proposed method can boost the performance for assessing news thumbnail representativeness, supporting the hypothesis. |
LinkBERT: Pretraining Language Models with Document Links (2022.acl-long)
Copied to clipboard
| Challenge: | Existing language model pretraining methods do not capture dependencies or knowledge that span across documents. |
| Approach: | They propose a language model pretraining method that leverages links between documents . they use masked language modeling and document relation prediction to model LMs . |
| Outcome: | The proposed method outperforms existing methods on downstream tasks across two domains. |
BhashaSutra: A Task-Centric Unified Survey of Indian NLP Datasets, Corpora, and Resources (2026.acl-long)
Copied to clipboard
| Challenge: | Existing reviews focus on a few high-resource languages or embed Indian languages within broad multilingual settings, limiting coverage of low-resourced and culturally diverse varieties. |
| Approach: | They present a unified survey of Indian NLP resources, covering 200+ datasets, 50+ benchmarks, and 100+ models, tools, and systems across text, speech, multimodal, and culturally grounded tasks. |
| Outcome: | The proposed survey covers 200+ datasets, 50+ benchmarks, and 100+ models, tools, and systems across text, speech, multimodal, and culturally grounded tasks. |
Outdated Issue Aware Decoding for Factual Knowledge Editing (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing knowledge editing methods retain outdated responses for reasoning questions . naively retraining LLMs can be computationally intensive and can lead to catastrophic forgetting . |
| Approach: | They propose a simple yet effective decoding strategy to enhance edited models on reasoning questions. |
| Outcome: | The proposed method outDates ISsue aware deCOding (DISCO) to improve models on reasoning questions. |
Pretrained Language Model Embryology: The Birth of ALBERT (2020.emnlp-main)
Copied to clipboard
| Challenge: | Pretraining of pretrained models (LMs) has been extensively studied, but what happened during pretraining is rarely studied. |
| Approach: | They propose to use a totipotent language model to study pretraining behavior . they find that linguistic knowledge and world knowledge do not generally improve as pretraining proceeds, nor do downstream tasks’ performance. |
| Outcome: | The model learns to reconstruct and predict tokens of different parts of speech (POS) in different learning speeds during pretraining. |
Coherence boosting: When your pretrained language model is not paying enough attention (2022.acl-long)
Copied to clipboard
| Challenge: | Long-range semantic coherence remains a challenge in automatic language generation and understanding. |
| Approach: | They propose a procedure that increases a model’s focus on a long context by distributional analyses of generated ordinary text and dialog responses. |
| Outcome: | The proposed procedure increases the model's focus on a long context. |
KazakhTTS2: Extending the Open-Source Kazakh TTS Corpus With More Data, Speakers, and Topics (2022.lrec-1)
Copied to clipboard
| Challenge: | Text-to-speech (TTS) is a process of converting written text into speech. |
| Approach: | They present an expanded version of their text-to-speech corpus for Kazakh . they propose to use the corpus to build high-quality TTS systems for the language . |
| Outcome: | The constructed corpus is sufficient to build robust TTS models for Kazakh and other Turkic languages, with a subjective mean opinion score ranging from 3.6 to 4.2 for all the five speakers. |
FlipDA: Effective and Robust Data Augmentation for Few-Shot Learning (2022.acl-long)
Copied to clipboard
| Challenge: | Existing methods for text data augmentation are limited to simple tasks and weak baselines. |
| Approach: | They propose a data augmentation method FlipDA that uses a generative model and a classifier to generate label-flipped data. |
| Outcome: | The proposed method improves many tasks while not negatively affecting the others. |
Large-Scale Transfer Learning for Natural Language Generation (P19-1)
Copied to clipboard
Sergey Golovanov, Rauf Kurbanov, Sergey Nikolenko, Kyryl Truskovskyi, Alexander Tselousov, Thomas Wolf
| Challenge: | Large-scale pretrained language models define state of the art in natural language processing, achieving outstanding performance on a variety of tasks. |
| Approach: | They propose to apply large-scale pretrained language models to natural language generation tasks by comparing architectural and training schemes. |
| Outcome: | The proposed architectures perform well on open-domain dialog as a typical high entropy generation task. |
Language-Codec: Bridging Discrete Codec Representations and Speech Language Models (2025.acl-long)
Copied to clipboard
Shengpeng Ji, Minghui Fang, Jialong Zuo, Ziyue Jiang, Dingdong Wang, Hanting Wang, Hai Huang, Zhou Zhao
| Challenge: | Existing gaps between discrete acoustic codecs and downstream speech language models . initial channel of codebooks contains excessive information, making it difficult to generate tokens from weakly supervised signals such as text. |
| Approach: | They propose a discrete acoustic codec for generating acustic tokens from weakly supervised signals. |
| Outcome: | The proposed language-codec outperforms competing audio compression algorithms and validates on downstream speech language models. |
CoDEx: A Comprehensive Knowledge Graph Completion Benchmark (2020.emnlp-main)
Copied to clipboard
| Challenge: | Knowledge graph completion benchmarks for knowledge graphs are often incomplete . however, the field has remained static over the past decade . |
| Approach: | They propose to use Wikidata and Wikipedia to improve on existing benchmarks . they analyze logical relation patterns, then perform baseline link prediction and triple classification . |
| Outcome: | The proposed datasets improve upon existing benchmarks in scope and difficulty. |
Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing benchmarks of social language are lacking for large language models. |
| Approach: | They propose a new benchmark that measures how well large language models understand social language by grouping 58 tasks into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
| Outcome: | The proposed model performs well at 58 tasks that are divided into five categories: humor & sarcasm, offensiveness, sentiment & emotion, and trustworthiness. |
GreekBART: The First Pretrained Greek Sequence-to-Sequence Model (2024.lrec-main)
Copied to clipboard
Iakovos Evdaimon, Hadi Abdine, Christos Xypolopoulos, Stamatis Outsios, Michalis Vazirgiannis, Giorgos Stamou
| Challenge: | Transfer learning has revolutionized the fields of Computer Vision and Natural Language Processing. |
| Approach: | They introduce a new language model, GreekBART, that is based on a BART-base architecture. |
| Outcome: | The proposed model outperforms BERT, GPT and other transformer-based models on discriminative tasks. |
Socratic Pretraining: Question-Driven Pretraining for Controllable Summarization (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to control document controllable summarization lack abundant labeled data. |
| Approach: | They propose a question-driven, unsupervised pretraining objective to improve controllability in document controllable summarization tasks. |
| Outcome: | The proposed method outperforms pre-finetuning approaches on QMSum and SQuALITY. |
Target-Aware Language Modeling via Granular Data Sampling (2024.emnlp-main)
Copied to clipboard
Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, Vikas Chandra
| Challenge: | Language model pretraining is the cornerstone of universal language models (LMs), creating generalpurpose representations to excel across a variety of downstream tasks. |
| Approach: | They propose to use multi-granular tokens to sample large-scale language models for domain-specific use cases. |
| Outcome: | The proposed model outperforms random sampled samples on eight benchmarks with 1% of the data and performs on par with the full RefinedWeb data. |
HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks (2023.findings-acl)
Copied to clipboard
| Challenge: | Pretraining and fine-tuning are the dominant paradigms in natural language processing. |
| Approach: | They propose a parameter-efficient multitask learning framework that takes trainable hyper-embeddings and visual modality as input and outputs weights for different modules in a pretrained language model. |
| Outcome: | The proposed framework adds fewer trainable parameters in multi-task learning while achieving superior performances and transfer ability compared to state-of-the-art methods. |
SPT: Learning to Selectively Insert Prompts for Better Prompt Tuning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Prompt tuning is a method of pre-trained models that optimizes the prompt to adapt to downstream tasks. |
| Approach: | They propose a framework that learns to select the proper prompt layers by inserting a probabilistic gate at each intermediate layer. |
| Outcome: | The proposed framework can perform better than the state-of-the-art prompt tuning frameworks on ten benchmark datasets. |
Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuning (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent pretrained language models extend from millions to billions of parameters. |
| Approach: | They propose a technique which forwards on a whole network while backwarding on resetting the gradients of the non-child network during the backward process. |
| Outcome: | The proposed technique outperforms the vanilla fine-tuning technique on various downstream tasks and can achieve better generalization performance by large margins. |
DeCoT: Debiasing Chain-of-Thought for Knowledge-Intensive Tasks in Large Language Models via Causal Intervention (2024.acl-long)
Copied to clipboard
| Challenge: | In large language models, external knowledge is required to augment their internal knowledge through prompts, but this does not guarantee that LLMs can identify and use relevant information in the prompts to conduct chain-of-thought reasoning. |
| Approach: | They propose a structural causal model to formally explain the internal knowledge bias of large language models (LLMs) they review the chain-of-thought (CoT) prompting from a causal perspective and find that biased information from pretrained models can impair LLMs’ reasoning abilities. |
| Outcome: | The proposed model enables more accurate CoT reasoning and enhances LLM generation on knowledge-intensive tasks. |
Improving Bengali and Hindi Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Bengali and Hindi are low-resource languages, and the state-of-the-art tokenization methods fail to separate roots from affixes. |
| Approach: | They used BERT and Wordpiece tokenizers to train a wordpiece tokenization system for Bengali and Hindi to model fine-grained character-level information. |
| Outcome: | The proposed tokenizers outperform the state-of-the-art and Wordpiece tokenizer for modeling Bengali and Hindi. |
A Diverse Set of Freely Available Linguistic Resources for Turkish (2023.acl-long)
Copied to clipboard
| Challenge: | despite the abundance of Turkish speakers, linguistic resources for natural language processing remain scarce. |
| Approach: | They propose a set of freely available linguistic resources for Turkish natural language processing . they provide corpora and pretrained models to help practitioners build their own applications . |
| Outcome: | The proposed linguistic resources are first of their kind and easy to use in a broad range of implementations. |
SAKE: Steering Activations for Knowledge Editing (2025.acl-long)
Copied to clipboard
| Challenge: | Large Langue Models memorize facts, but they suffer from several limitations . Using a single input prompt is insufficient to capture the complexity of the knowledge scope affected by edits. |
| Approach: | They propose a steering activation method that models a fact to be edited as a distribution rather than a single prompt. |
| Outcome: | The proposed method can perform more robust edits than existing methods. |
MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing pretrained language models for mental health detection are inadequate . one in four people worldwide suffers from mental disorders . |
| Approach: | They train and release two pretrained masked language models to benefit machine learning for mental healthcare research . they demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks. |
| Outcome: | The proposed models improve mental health detection tasks on several benchmarks and are available for free. |
Efficient Active Learning with Adapters (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing studies show that distilled versions of pretrained models are not always available. |
| Approach: | They propose to use distilled versions of successor models as acquisition models to reduce the training cost of the model. |
| Outcome: | The proposed approach reduces the training cost of the model and does not cause the acquisition-successor mismatch (ASM) problem. |
HypoNLI: Exploring the Artificial Patterns of Hypothesis-only Bias in Natural Language Inference (2020.lrec-1)
Copied to clipboard
| Challenge: | Recent studies have shown that for models trained on datasets for natural language inference (NLI), it is possible to make correct predictions by looking at the hypothesis while completely ignoring the premise. |
| Approach: | They propose to derive adversarial examples in terms of the hypothesis-only bias and explore eligible ways to mitigate such bias. |
| Outcome: | The proposed models can be used to mitigate the hypothesis-only bias by using down-sampling and adversarial training. |
ConKI: Contrastive Knowledge Injection for Multimodal Sentiment Analysis (2023.findings-acl)
Copied to clipboard
Yakun Yu, Mingjun Zhao, Shi-ang Qi, Feiran Sun, Baoxun Wang, Weidong Guo, Xiaoli Wang, Lei Yang, Di Niu
| Challenge: | Existing methods for multimodal sentiment analysis focus on general knowledge, which is inadequate to identify specific sentiments across modalities. |
| Approach: | They propose a method where specific-knowledge representations for each modality can be learned together with general knowledge representations via knowledge injection based on an adapter architecture. |
| Outcome: | The proposed method outperforms all prior methods on three popular benchmarks on multimodal sentiment analysis metrics. |
Enhancing Text Editing for Grammatical Error Correction: Arabic as a Case Study (2025.acl-long)
Copied to clipboard
| Challenge: | Text editing is a wellstudied problem for grammatical error correction (GEC) but it is not the most efficient for morphologically rich languages like Arabic. |
| Approach: | They propose a text editing approach that derives edit tags directly from data, eliminating the need for language-specific edits. |
| Outcome: | The proposed approach achieves SOTA results on Arabic and performs on par with SOTA on two other languages. |
Characterizing Text Datasets with Psycholinguistic Features (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing algorithms for fine-tuning language models on task-specific data are not optimal for all scenarios. |
| Approach: | They propose a framework to fine-tune text models on task-specific data using meta-datasets. |
| Outcome: | The proposed framework evaluates multiple dimensions of text and discourse, producing interpretable, low-dimensional embeddings. |
Difference-Masking: Choosing What to Mask in Continued Pretraining (2023.findings-emnlp)
Copied to clipboard
Alex Wilf, Syeda Akter, Leena Mathur, Paul Liang, Sheryl Mathew, Mengrou Shou, Eric Nyberg, Louis-Philippe Morency
| Challenge: | Existing approaches to masked prediction have shown that deciding what to mask can substantially improve learning outcomes. |
| Approach: | They propose a masking strategy that automatically chooses what to mask during continued pretraining by considering what makes a task domain different from the pretraining domain. |
| Outcome: | The proposed masking strategy outperforms baselines on language-only and multimodal video tasks. |
Advancing Vision-Language Models with Adapter Ensemble Strategies (2024.findings-emnlp)
Copied to clipboard
| Challenge: | CLIP revolutes vision-language pretraining by using contrastive learning on paired web data. |
| Approach: | They propose to combine a "adapter ensemble" with traditional machine learning techniques to augment large-scale pretrained vision-language models. |
| Outcome: | The proposed model outperforms baselines and derives improvement when the number of ensemble parameters increases. |
DocSplit: Simple Contrastive Pretraining for Large Document Embeddings (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing model pretraining methods only consider local information, resulting in low-quality embeddings for large documents. |
| Approach: | They propose a new method which forces models to consider the entire global context of a large document. |
| Outcome: | The proposed method outperforms existing models on document classification, few shot learning, and retrieval tasks. |
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Considerable efforts have been and are still being put into increasing the context length of Large Language Models (LLMs) |
| Approach: | They propose an approach that divides long contexts into chunks, compresses each into soft prompts using a pretrained text encoder, and aligns these representations with a decoder-only LLM via an adapter. |
| Outcome: | The proposed approach outperforms 8 state-of-the-art methods in effectiveness and efficiency for document summarization and question answering, and achieves the best performance on LongBench v2 among models of comparable size. |
CoCo: Coherence-Enhanced Machine-Generated Text Detection Under Low Resource With Contrastive Learning (2023.emnlp-main)
Copied to clipboard
| Challenge: | Recent proposed methods fail to consider the linguistic structure of texts and lack the ability to handle the low-resource problem. |
| Approach: | They propose a coherence-based contrastive learning model named CoCo to detect MGTs under low-resource scenario. |
| Outcome: | The proposed model outperforms state-of-the-art methods on two datasets and two self-constructed datasets. |
Multimodal and Multilingual Laughter Detection in Stand-Up Comedy Videos (2024.lrec-main)
Copied to clipboard
| Challenge: | Using TED talks, we use laughter detection software to capture humor in the sitcom genre. |
| Approach: | They develop a multimodal multilingual dataset in Russian and English with a particular emphasis on laughter detection techniques. |
| Outcome: | The proposed model outperforms peak detection and machine learning, while the latter shows promise and warrants further study. |
ApiQ: Finetuning of 2-Bit Quantized Large Language Model (2024.emnlp-main)
Copied to clipboard
| Challenge: | Memory-efficient finetuning of large language models (LLMs) has attracted huge attention with the increasing size of LLMs due to the constraints posed by GPU memory limitations and the effectiveness of these methods compared to full finetune. |
| Approach: | They propose a memory-efficient finetuning framework called ApiQ to restore lost information from quantization by initializing LoRA components and quantizing weights of LLMs. |
| Outcome: | The proposed framework maintains the original LLM’s activation precision while mitigating error propagation from shallower into deeper layers. |
Just KIDDIN’ : Knowledge Infusion and Distillation for Detection of INdecent Memes (2025.findings-acl)
Copied to clipboard
| Challenge: | Existing approaches to detect toxicity in online multimodal environments require common-sense reasoning and contextual awareness. |
| Approach: | They propose a hybrid neurosymbolic framework that unifies distillation of implicit contextual knowledge from Large Vision-Language Models and infusion of explicit relational semantics through sub-graphs from Knowledge Graphs. |
| Outcome: | The proposed framework outperforms state-of-the-art models on two datasets with improvements of 0.5%, and 10.6% in HatefulMemes Benchmark. |
Anatomy of Unlearning: The Dual Impact of Fact Salience and Model Fine-Tuning (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing studies assume that all facts are equally forgettable . popular facts, frequent and widely distributed, may be more deeply embedded than rare ones, making them harder to erase. |
| Approach: | They propose a benchmark to evaluate how unlearning differs between pretrained and supervised fine-tuned models when fact popularity is taken into account. |
| Outcome: | The proposed model is compared with pretrained and SFT models on the forget data and shows that it performs better on both models. |
ProcrustesGPT: Compressing LLMs with Structured Matrices and Orthogonal Transformations (2025.findings-acl)
Copied to clipboard
| Challenge: | Structured matrix representations of large language models are a promising way to reduce the number of parameters in natural language processing tasks but require a significant amount of computational and memory resources. |
| Approach: | They propose to utilize invariance of the network output under certain orthogonal transformations of weight matrices to identify transformations that improve compressibility of weights within structured classes. |
| Outcome: | The proposed approach is applicable to various types of structured matrices that support efficient projection operations. |
Segment, Embed, and Align: A Universal Recipe for Aligning Subtitles to Signing (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches for aligning spoken language text to sign language videos rely on end-to-end training tied to a specific language or dataset. |
| Approach: | They propose a universal approach for aligning spoken language text with corresponding timestamps to sign language videos using a lightweight dynamic programming procedure. |
| Outcome: | The proposed method can be used on four sign language datasets and is highly efficient on CPU. |
Why LoRA Fails to Forget: Regularized Low-Rank Adaptation Against Backdoors in Language Models (2026.findings-acl)
Copied to clipboard
| Challenge: | Low-Rank Adaptation (LoRA) is widely used for parameter-efficient fine-tuning of large language models, but is ineffective at removing backdoor behaviors from poisoned pretrained models when fine-timing on clean datasets. |
| Approach: | They propose a low-rank Adaptation method which increases spectral strength and corrects alignment through clean-strengthened regularization and trigger-insensitive constraints. |
| Outcome: | The proposed method significantly reduces attack success rates while maintaining clean accuracy. |
Learning to Control Summaries with Score Ranking (2026.findings-acl)
Copied to clipboard
| Challenge: | Recent advances in summarization focus on improving summary quality across multiple dimensions, but they overlook the challenge of controlling summary generation with respect to individual dimensions. |
| Approach: | They propose a loss function that aligns model outputs with fine-grained, model-based evaluation scores to enable both improvement in summary quality and dimension-specific control. |
| Outcome: | The proposed method improves the overall quality of summaries while maintaining strong control over individual quality dimensions. |
MTP-RL: Acceleration of Reinforcement Learning Rollouts with Policy-Aligned Multi-Token Prediction (2026.findings-acl)
Copied to clipboard
| Challenge: | Reinforcement learning (RL) is widely applied to boost the performance of pretrained models, yet its training efficiency is severely constrained by rollout generation. |
| Approach: | They propose a framework that accelerates the rollout phase for diverse models by equipping a pipeline to equip the multi-layer parameter-sharing MTP for all models and an advantage-aware MTP optimization strategy. |
| Outcome: | The proposed framework achieves stable growth of acceptance length during RL training, and also accelerates RL rollouts, achieving an average 23.1%–55.3% reduction in rollout time compared to baselines. |
PseudoSeer: a Search Engine for Pseudocode (2026.findings-acl)
Copied to clipboard
| Challenge: | PseudoSeer is a search engine for academic pseudocode that indexes over 320,000 implementations extracted from 2.2 million arXiv papers. |
| Approach: | They propose to use caption-reference pairs to match short queries with a median length of five words against long documents composed primarily of natural language with limited LaTeX notation. |
| Outcome: | The proposed algorithm outperforms the best pretrained model by 8.7 points and achieves 66.5% R@10 . |
Leveraging Pretrained Language Models as Energy Functions for Glauber Dynamics Text Diffusion (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for generating text using Glauber dynamics are autoregressive, but they face a number of limitations. |
| Approach: | They propose a discrete diffusion-based generative model for text generation using Glauber dynamics from statistical physics and use pretrained causal/masked language models to improve the quality of the generated text. |
| Outcome: | The proposed model outperforms existing models on some common sense reasoning tasks and planning/search tasks. |